Storage of a Document Using Multiple Representations

ABSTRACT

Some embodiments provide a method for storing a document. The method stores a content stream representation of the document that includes an ordered stream of code representations for primitive elements of the document. Each code representation of a primitive element has an index that indicates the order in the content stream of the primitive element representation. The method stores an object representation of the document that includes a set of object nodes arranged in a tree structure. Each object node references a range of indices in the content stream.

CLAIM OF BENEFIT TO PRIOR APPLICATIONS

This application claims the benefit of U.S. Provisional Application61/433,951, entitled “Reconstruction of Lists, Ordering Page Content,Storing Multiple Representations for a Document, and Adaptive GraphicObjects”, filed Jan. 18, 2011, and U.S. Provisional Application61/433,963, entitled “Storage of a Document Using MultipleRepresentations”, filed Jan. 18, 2011. U.S. Provisional Applications61/433,951 and 61/433,963 are incorporated herein by reference.

BACKGROUND

Electronic document file formats and in-memory representations used bysoftware that process the documents will generally be either streamrepresentations or object representations. Stream representationsgenerally consist of a sequence of character codes or other primitivedata elements into which are interspersed special non-character valuesor sequences of values that signal a change in state or context (e.g.,text style, transform or drawing properties, or the beginning or end ofa mode of parsing). Software that processes the data in such a streammust start at the beginning and visit each element in turn in order todetermine the state reached at a given data element, since this state isthe net result of all changes occurring up to that point.

On the other hand, object representations consist of a collection ofobjects representing component parts of the document. The objects cancontain object properties, pointers or references to other, relatedobjects, and a portion of the content of the document. For example, aparagraph object might include a line spacing property and the text ofthe paragraph, while a section object might include an orderedcollection of pointers to all the paragraph objects and illustrationobjects that comprise that section. Often object representations areprimarily hierarchical, the graph of object references forming a tree(e.g. documents contain pages which contain zones which can containother zones and layout areas which contain columns which containparagraphs). However, even in the case of hierarchical models, there maybe additional object references separate from the tree graph (e.g. zonesmay contain references to the shared graphic objects that contribute totheir boundary and the boundary of other zones, while paragraphs maycontain references to shared text style objects).

Applications often choose a different representation for their fileformat than their in-memory representation. While each of the formats isefficient for certain operations, each format is also inefficient forother operations. For instance, while a stream representation is usefulfor string searches and other processing that does not depend ondetailed knowledge of object properties, the stream representation willnot be useful for accessing a particular object.

BRIEF SUMMARY

Some embodiments of the invention provide a novel method for storing adocument that stores at least two different representations of thedocument. In some embodiments, the method stores a stream representationof the document and one or more object representations. Theserepresentations may be stored as files or as in-memory representationsof the document.

In some embodiments, the stream representation contains only purecontent arranged in an order (e.g., a sequence of character codes fordocument text). An object representation is stored as a tree of nodesthat represent groupings of content and associated information about thegroupings (e.g., document structures, styles, etc.). Rather than storingcontent in the nodes, each node contains a reference to index positionsin the stream representation.

Some embodiments include two object representations—one structure treeand one styling tree. The structure tree stores the documentstructure—i.e., the sections, pages, layouts, columns, paragraphs, textlines, words, etc. This information may be derived from informationencoded in the document (e.g., in a rich-text word processing document)or derived from a reconstruction of a document that does not have suchinformation encoded (e.g., a vector graphics document such as a PDFfile). Each structure node in the tree references a range of indices inthe stream at which the content for the structure is found. In such atree, a parent node represents a structure containing the structurerepresented by a child node.

The styling tree of some embodiments stores information about theformatting of the characters in the document. Some embodiments store ahierarchy of formatting traits, with each node in the object treerepresenting a format style and a range of indices in the stream. Forinstance, an underlined section within a larger bold section would berepresented by a bold node with a range of the entire bold section. Thebold node would have a child underline node with a range of theunderlined section within the bold section. In such a tree, a parentnode represents a style property inherited by the child node.

In addition to structure and styling trees, some embodiments may storesother different types of object representations for a document. Forinstance, different embodiments will store an object tree of clip pathsfor drawing a document, revision trees that track changes to a documentby different authors/editors, semantic document trees, etc.

This multi-representation approach enables efficiency for various commondocument actions. Selection of a structure (e.g., a word, line,paragraph, etc.) can be efficiently performed by navigating thestructure tree to find the desired structure and then using the indicesto identify the content to select in the stream. Other examples ofefficiently performed actions include certain navigation operations(e.g., finding a search term in a chapter heading) and conversion of adocument from one format to another (e.g., in order to switch betweenapplications).

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.Moreover, the claimed subject matters are not to be limited by theillustrative details in the Summary, Detailed Description and theDrawing, but rather are to be defined by the appended claims, becausethe claimed subject matters can be embodied in other specific formswithout departing from the spirit of the subject matters.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates a document for which multiple differentrepresentations are created and stored according to some embodiments.

FIG. 2 illustrates an example of a page of a document.

FIG. 3 illustrates a stream representation of some embodiments for thepage of FIG. 2.

FIG. 4 illustrates an object representation for the page of FIG. 2.

FIG. 5 conceptually illustrates a column object of some embodiments.

FIG. 6 illustrates a single-page document.

FIG. 7 conceptually illustrates two separate object models for thedocument of FIG. 6.

FIG. 8 illustrates an example of a styling object tree in which eachlevel of the hierarchy stores information about a particular stylingproperty.

FIG. 9 illustrates a word selection process of some embodiments.

FIG. 10 illustrates the page of FIG. 2 with a word highlighted.

FIG. 11 conceptually illustrates a process of some embodiments foridentifying a desired object in a document structure tree thatrepresents a structure in the document to which a user wishes tonavigate.

FIG. 12 illustrates searching for a particular chapter title in ane-book.

FIG. 13 conceptually illustrates a process for using the dualrepresentation data model to convert a selection of a portion of thedocument into another format.

FIG. 14 illustrates a selection in an example document page.

FIG. 15 illustrates an example object tree for the document page of FIG.14 and an example output stream in a new format produced by theconversion process using the object tree.

FIG. 16 a-16 b conceptually illustrates an example recursive processthat can be performed as part of the process of FIG. 13.

FIG. 17 conceptually illustrates an electronic system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following description, numerous details are set forth for purposeof explanation. However, one of ordinary skill in the art will realizethat the invention may be practiced without the use of these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order not to obscure the description of theinvention with unnecessary detail.

Some embodiments of the invention provide a novel method for storing adocument that stores at least two different representations of thedocument. In some embodiments, the method stores a stream representationof the document and one or more object representations. Theserepresentations may be stored as files or as in-memory representationsof the document.

FIG. 1 illustrates a document 100 for which multiple differentrepresentations are created and stored according to some embodiments. Asshown, in this example three different representations are stored forthe document. The first representation is stream representation 105. Insome embodiments, the stream representation contains only pure contentarranged in an order (e.g., a sequence of character codes for documenttext). The stream representation 105 is illustrated as a sequence ofcharacters strung together, though in some embodiments the stream isactually stored as a sequence of bytes (i.e., 0s and 1s) representinghexadecimal character codes.

The second and third representations for document 100 are objectrepresentations 110 and 115. In some embodiments, an objectrepresentation is stored as a tree of nodes that represent groupings ofcontent and associated information about the groupings (e.g., documentstructures, styles, etc.). Rather than storing content in the nodes,each node contains a reference to index positions in the streamrepresentation.

As shown in FIG. 1, some embodiments include two (or more) objectrepresentations (e.g., a structure tree and a styling tree). Thestructure tree of some embodiments stores the document structure—i.e.,the sections, pages, layouts, columns, paragraphs, text lines, words,etc. This information may be derived from information encoded in thedocument (e.g., in a rich-text word processing document) or derived froma reconstruction of a document that does not have such informationencoded (e.g., a vector graphics document such as a PDF file). Eachstructure node in the tree references a range of indices in the streamat which the content for the structure is found. In such a tree, aparent node represents a structure containing the structure representedby a child node.

The styling tree of some embodiments stores information about theformatting of the characters in the document. Some embodiments store ahierarchy of formatting traits, with each node in the object treerepresenting a format style and a range of indices in the stream. Forinstance, an underlined section within a larger bold section would berepresented by a bold node with a range of the entire bold section. Thebold node would have a child underline node with a range of theunderlined section within the bold section. In such a tree, a parentnode represents a style property inherited by the child node.

In addition to structure and styling trees, some embodiments may storesother different types of object representations for a document. Forinstance, different embodiments will store an object tree of clip pathsfor drawing a document, revision trees that track changes to a documentby different authors/editors, semantic document trees, etc.

This multi-representation approach enables efficiency for various commondocument actions. Selection of a structure (e.g., a word, line,paragraph, etc.) can be efficiently performed by navigating thestructure tree to find the desired structure and then using the indicesto identify the content to select in the stream. Other examples ofefficiently performed actions include certain navigation operations(e.g., finding a search term in a chapter heading) and conversion of adocument from one format to another (e.g., in order to switch betweenapplications).

Certain operations are most efficiently performed using the structureobject tree (e.g., identifying a user selection from start and end pointcoordinates on a page). Other operations are more efficiently performedusing the style tree (e.g., determining the net style propertiesapplicable to a given character, while yet other operations are mostefficiently performed using the stream representation (e.g., indexingthe document for a full-text search) or using different object treesstored for the document.

Several more detailed embodiments of the invention are described below.Section I describes a data model for a document that includes a contentstream and a document structure tree. Section II then describes theaddition of a style properties tree as well as other object trees to thedata model for a document. Section III describes various uses of thedata model. Finally, Section V describes an electronic system with whichsome embodiments of the invention are implemented.

I. Multiple Representations for a Document

As mentioned above, some embodiments store multiple representations fora document. In some embodiments, an application stores (i) a streamrepresentation and (ii) one or more object representations. The multiplerepresentations are complimentary in that both can be accessedsimultaneously and provide different types of information that can beused together for more efficient document processing. In someembodiments, the object representations include a structure object treethat stores information about document structures (e.g., pages, layouts,columns, paragraphs, text lines, words, etc.). In some embodiments, thisstructure tree is derived in a manner similar to that described in theUnited States Patent Publication No. 2010/0174985, entitled“Identification of Layout and Content Flow of an Unstructured Document”.

FIG. 2 illustrates an example of a page 200. The page 200 might be asingle-page document or a page from a multi-page document. As shown inthe figure, the page 200 includes a single column with three paragraphs205-215. The first paragraph 205 includes five text lines, the secondparagraph 210 includes six text lines, and the third paragraph 215includes five text lines. Each text line includes various differentnumbers of words.

FIG. 3 illustrates a stream representation 300 of some embodiments forthe page 200. As shown, the stream representation 300 is a string ofhexadecimal character codes representing the content of page 200. Eachcharacter in the document is converted into the hexadecimal code used torepresent the character (e.g., 42 in hexadecimal for “B”). In thisexample, spaces and line breaks are also included in the stream, thoughin some embodiments these white space characters are not strictlynecessary and can be deduced from an object representation of the page(e.g., a structure tree).

The illustrated stream uses ASCII (2 hex digit) representations for thecharacters, while some embodiments use Unicode (4 hex digit)representations. Though not included in the example page 200, someembodiments represent non-character primitive content (e.g., images,shapes, etc.) with a placeholder character (e.g., the Unicode “objectreplacement character” with hexadecimal code FFFC). While the charactersare shown as hexadecimal codes, some embodiments store the string as aseries of bits (i.e., 0s and 1s) representing the hex code or use adifferent code for the characters in a page.

FIG. 4 illustrates an example object representation 400 of someembodiments for the page 200. Specifically, the object representation400 is a portion of a structure tree that stores an object for eachstructure in a document. FIG. 4 illustrates the portion of the documentstructure tree for the page 200. In some embodiments, a single parentnode for the document would include one such portion for each page inthe document.

Each node in the document structure tree 400 is shown as a range ofnumbers. These numbers represent references to indices in the contentstream representation 300. As shown, the illustrated portion of thedocument object tree 400 includes a page node 405, a single column node410, three paragraph nodes 415-425, and sixteen text line nodes 431-446.While only two word nodes 450 and 455 are shown in this figure, one ofordinary skill will recognize that many word nodes (more than can beshown in the diagram) are stored in the object tree. In addition, belowthe word nodes may be primitive character nodes (e.g., glyphs), whicheach reference a single character in the stream.

In some embodiments, this structure information is extracted frominformation encoded in the document when such information is included(e.g., in a rich-text word processing document such as that created byApple Pages® or Microsoft Word®. When the document does not include suchencoded structure information (e.g., a typical PDF document, someembodiments use a document reconstruction process that derives thestructure information based on the primitive elements (e.g., glyphs) inthe document. The document reconstruction process of some embodimentsidentifies text lines, columns, paragraphs, a reading order through thedocument elements, etc.

In the structure tree 400, the ranges for a particular structureindicate the range in the content stream to which that structurecorresponds. Thus, for instance, the first paragraph 415 corresponds tohexadecimal code characters 1-338. In some embodiments, this informationis stored as a pair of numbers. Some embodiments store the start indexand the count of subsequent consecutive indices in the range, whileother embodiments store the start and end indices. FIG. 4 illustratesthe range using the start and end indices for simplicity.

In some embodiments, as mentioned, the content stream includes separatorcharacters (e.g., spaces, carriage returns, etc.), even though these canbe derived from the object tree (e.g., a space between words, a carriagereturn between paragraphs). Some embodiments include these separatorcharacters within the index range of the objects in the structure tree.For instance, the range of paragraph 415 includes the hex code for thecarriage return at the end of the first paragraph.

In addition to the ranges shown in the figure, in some embodiments eachstructure node (i.e., structure object) includes additional informationabout the structure. FIG. 5 conceptually illustrates a column object 500of some embodiments. This object stores information about the columndefined by the object. In some embodiments, the column object 500 is anobject in a column class, and the various data shown are variables ofthe object. As shown, the column object 500 includes a unique column IDthat is used to identify the column, an index range that stores a rangeof indices pointing into the content stream for the document containingthe column, and a set of column attributes 505. In addition, someembodiments include references to the children of the object (in thiscase, several paragraphs) as well as a reference to the parent of theobject (in this case, a page). Some embodiments only store references inone direction for all objects in a structure tree (i.e., references toeither an objects parent or its children).

The column attributes 505 store various information about the column asa structure: the bounds of the column within the page, the rotationangle of the column (i.e., whether the column is oriented horizontallyor vertically on the page, or somewhere in between), references to theprevious and next column in reading flow through the document, and otherattributes of the column. In some embodiments, different structureobjects include different attributes. For instance, a paragraph objectmight include similar attributes (bounds, rotation angle, next andprevious paragraph) to the column object 500, but could also includeline spacing information, alignment information, etc.

In some embodiments, five conditions are imposed on the stream andobject trees to ensure proper coordination between the differentrepresentations. First, the stream is arranged in a natural order forefficient stream processing. In the case of an electronic document onwhich the relevant stream processing includes text search and selection,the natural order is reading order. Reading order may be derived frominformation encoded in the document (e.g., by a word-processingapplication) or may come from an analysis of the document as describedin US Patent Publication No. 2010/0174985, which is incorporated byreference above.

Second, the index positions referenced by each node lie in a singlecontiguous range R={i, i+1, . . . , i+n−1}. That is, each objectreferences a range of indices that has no gaps in the content stream.Thus, the range may be stored as a pair of whole numbers (i, n), where iis the start index and n is the count of consecutive indices in therange. As shown in FIG. 4, the range could also be stored as (i, i+n−1),using the start and end indices.

Third, the range of a parent node is the union of the ranges of itsnodes. This prevents gaps in a level of the document model. Forinstance, the layouts on a page must include all of the primitiveelements of the page. The columns in a layout must include all of theprimitive elements of the layout, the paragraphs in a column mustinclude all of the primitive elements of the column, etc. In someembodiments, however, the range of a parent node merely contains theunion of its child nodes. That is, a child node cannot include anycontent not in its parent node, but a parent node may contain contentnot in any of its child nodes. In such embodiments, there may be gapsbetween child nodes (e.g., for spaces, etc.). Processes that require thefull union can effectively deduce nodes for the missing gaps as theyoperate. Thus, nodes that do not add any information to the data modelneed not be stored in these embodiments, which frees up storage space.

Fourth, the ranges of any two sibling nodes are disjoint. This meansthat there is no overlap between two sibling nodes. Thus, a charactercannot be in two different words, lines, paragraphs, etc. Lastly, if afirst node is a prior sibling of a second node, then the start index ofthe first node is less than or equal to the start index of the secondnode. This mandates that the sibling nodes be placed in order. Thus, asshown, the node 431 for the first text line is the first node under theparagraph node 415. As is apparent, the document structure tree 400 andstream 300 satisfy these five properties. The result, for the documentstructure tree 400, is that each level of the hierarchy includes allcontent on the page 200.

II. Additional Object Representations

The document structure tree is one type of object representation thatmay be used by some embodiments in conjunction with a content stream inorder to represent a document. In addition to (or instead of) thedocument structure tree, some embodiments also include additional objectrepresentations of a document. One such object representation implementscontent styling, with references to the same content stream as thedocument structure tree. In order to draw the document, a softwareapplication may recursively descend through the nodes of the style treein order to resolve inherited styles with which to draw content. Ratherthan setting style properties for each character, using a style objecttree allows each node in the tree to be re-used by each of its children.

Certain professional style models, such as eXtensible StylesheetLanguage (XSL), include the ability to lay out text in an orderdifferent from the reading order of the text. For instance, a tablecaption might be deemed to be prior to the table in reading order, basedon reading order rules, but is drawn below the table. In XSL, this isviewed as transforming the structure tree into a “formatting object”tree, in which the formatting objects include layout and styleinformation. To implement such a tree in the multiple representationcontext, some embodiments require removing the fifth condition describedin the previous section, and allowing siblings to be out of order.

Even in the case where the formatting nodes are out of order, both theformatting nodes and the structure nodes (e.g., those in structure tree400) retain ranges in a common content stream. Thus, an application canquickly determine which structure objects give rise to each formattingobject. If a formatting object has a range of i to j, then any structureobjects whose range intersects that range are affected by the formattingobject.

FIG. 6 illustrates a single-page document 600. The document 600 includesthree paragraphs, each with numerous words. In addition, variousformatting traits are applied to portions of the first paragraph. Aportion of the first line is italicized, while portions of the secondline have italic, bold, and underlined text. Both the third and fourthlines also include applied styling changes. As the text of document 600is the same as that of page 200, the content stream 300 accuratelyrepresents the document 600.

FIG. 7 conceptually illustrates two separate object models for thedocument 600. The first object model 705 is a document structure treesimilar to document structure tree 400. In this and subsequent figures,character ranges (i.e., counts of the characters in the document) ratherthan indices of content stream hex code are used for simplicity. One ofordinary skill will recognize that these ranges can be easily convertedinto content stream indices. Also for simplicity, the document structuremodel shows only the page, paragraphs, and word nodes. One of ordinaryskill will recognize that different embodiments will use differentlevels of hierarchical structure objects to represent the documentstructure—text lines may be included between paragraphs and words asshown in the structure tree 400 of FIG. 4, columns may be includedbetween paragraphs and pages, etc.

The second object model 710 is a style properties tree. Rather thanorganizing content objects, the style properties tree organizesformatting properties of content, irrespective of the structure to whichthe content belongs. The same data model properties that apply to thedocument structure tree, discussed above in Section I, also apply tostyle properties tree 710. That is, (i) the index positions referencedby each node are a contiguous range, (ii) the range of a parent node isthe union of the ranges of its child nodes, (iii) the ranges of any twosibling nodes are disjoint, and (iv) the start indices of sibling nodesare ordered.

All of the characters have the same font face (e.g., Helvetica, TimesNew Roman, etc.) and size (e.g., 12 pt.), so the root node 715 includesthese properties and its range is all 674 characters. Some embodimentswill include several tree structures such as that shown in FIG. 7, witha higher-level root node for the page. In addition, the font face andthe size may be defined as different levels of objects, in case anentire document is in a particular font but different sections havedifferent sizes. The children (and grand-children, etc.) of the rootnode 715 apply various formatting properties to the referenced sets ofcharacters. For instance, the italicized beginning of the document isindicated by the node 720, which indicates the italicization of thefirst 28 characters of the document. As no additional formatting isapplied to these characters, the node does not have any child nodes.

The italics node 725, on the other hand, has additional child nodes todifferentiate the underlined portion from the portion withoutunderlining. As shown, of the characters 42-61 that are italicized,three of these characters 42-44 are not underlined while the remainingcharacters 45-61 are underlined. In addition, within the underlinedcharacters, characters 45-54 are bolded, while characters 55-61 have aregular weight (i.e., are not bolded). Thus, to draw character 53, anapplication would traverse the tree 710 and determine that it has thefont face and size applied to the entire document, is italicized (node725), underlined (node 730), and bolded (node 735). In this manner, eachnode inherits the formatting properties of its parents.

In order to maintain the continuity properties of an object model, someembodiments include non-change nodes, such as node 740 or node 745. Node740 indicates that no formatting applies to characters 29-41 other thanthat of the root node. This node serves to fill the gap between theitalics node 720 and the italics node 725. The node 745 indicates thatcharacters 55-61 are not bolded. These characters, nevertheless, retainthe underline property inherited from node 730 and the italics propertyinherited from node 725, as well as the font face and size from node715.

Some embodiments, rather than having a style formatting tree withapplied properties such as bold, italics, and underline all having thepossibility of being on the same level in the tree, include specificlevels for specific properties. For instance, the tree might have afirst level indicating font face, a second level indicating font size, athird indicating weight (e.g., bolded or not bolded, along withgradations), a fourth indicating slant (e.g., italicized or straight,with gradations), a fifth indicating underline properties (e.g., nounderline, single underline, double underline, etc.), as well as levelsfor other character format properties.

FIG. 8 illustrates an example of such a styling object tree 800, inwhich each level of the hierarchy stores information about a particularstyling property. The object tree 800 includes hierarchical levels forfont face, font size, slant, font weight, and underlining. The styleobjects refer to the same document 600, and thus conveys the sameinformation as styling tree 710. In the case of object tree 800, eachlevel of the hierarchy refers to the full range of indices in thecontent stream, and the objects satisfy the data model propertiesdescribed above in Section I.

As conveyed by the objects in the top two levels, font face object 805and font size object 810, the font face (Helvetica) and font size (13pt.) are the same throughout the document. The next level of hierarchyis the slant, which in this case indicates either no slant oritalicized. Below the slant objects are font weight objects, and thenunderlining objects. In different embodiments, the different propertiesmay be arranged in different orders in the hierarchy, as there is nonatural hierarchy to the font styling properties in the way that such anorder exists for structures. Some embodiments use a dynamic orderdetermined based on what will make a tree more simple (e.g., if there isa large bold section that is intermittently italicized, have the slantobjects as children of the font weight objects).

One reason for keeping the formatting object tree separate from thecontent tree is so that a continuous stretch of formatted text (e.g., anitalicized section that runs from one column to the next) need not besplit among multiple objects as far as applying the styling. In thiscase, a single italic style object could not be a child of both columnswithout violating the tree structure rules, and thus a separate styletree is useful. In the case shown in FIG. 6, the italics that run fromthe end of the first line to the middle of the second line would need tobe represented by two separate objects in order to preserve ahierarchical structure.

Since both the structure tree nodes and the formatting object tree nodesretain ranges in a common content stream, an application can easilydetermine which structure objects give rise to each formatting object(the structure tree objects whose ranges overlap the range of theformatting object). Thus, using a separate style object tree may beespecially useful for implementing constraint-based styling, in whichconstraints are imposed on the allowed formatting. Examples of suchconstraints include a minimum amount of text that must be present on apage, a requirement that a collection of content fit within a fixed-sizebox, or that a diagram be within a minimum distance of an article thatreferences it. Until the formatting is complete, an application cannotidentify whether such a constraint is violated (e.g., text is larger ifbolded and thus might not fit entirely within a fixed-size box). If theapplication knows which structure objects gave rise to a formattingobject that caused a violation, the application can automatically adjustthe styling of these structures until the constraint is satisfied. Forinstance, the relevant text can be compressed or images reduced in sizeuntil the resulting objects fit where required after the application ofthe formatting using the style properties tree.

In the example of XSL, when performing a transformation between twotrees, there is no information kept about how newly created objects inthe second tree relate to the old objects in the first tree—instead,there is simply a procedure that matches an old object to a new object.When performing transformations with the data structures describedherein, some embodiments define an additional attribute in anothernamespace to keep track of the index range of an object. A formatter canthen be aware of the indices for an object and output the indices in XSLformatting objects using the same namespace (or just store all theinformation in memory). In some embodiments, the transformation is doneusing XSL Transformation language (XSLT).

While the above sections describe two specific types of object tree (thedocument structure tree and the style properties tree, one of ordinaryskill will recognize that these models are merely two examples of themany different types of object trees that can be stored with nodespointing into the content stream. One example of such an object tree isa clip path tree that stores a set of clip paths for drawing a document.A clip path is a path defined such that content inside the path is drawnwhile content outside the path is ignored (i.e., a cutout). Forinstance, a page will have a clip path (thereby preventing content thatruns off the page from being drawn). Within the page, a text box mighthave a clip path, and an image in a frame inside the text box, etc.These clip paths (which may change dynamically) can be arranged into atree structure that references content being drawn inside the clip path.

Another example is a high-level semantic document tree that is unrelatedto layout, as compared with the object structure tree that defines thelayout (e.g., giving bounds for the objects). Such a document tree mightinclude nodes such as article objects (that may span multiple pages),chapter header nodes, bibliographic entry nodes (which could containtitle nodes, author nodes, etc.). In such a semantic tree, a paragraphobject might span multiple columns or pages (and there might not becolumn objects or page objects as these are layout constructs ratherthan semantic constructs). The content can then be laid out in a varietyof different layouts (e.g., all on one page, spread across multiplepages, using different numbers of columns, using different fonts andother stylings, etc.) based on user, device, or application settings.DocBook is an example of an XML format that has a similar structure. Thenodes of the semantic document tree could point into the content streamjust as do the nodes of the content structure tree.

Additional such object trees can exist as well. For instance, revisionstrees that track revisions of a document by different authors,annotations trees that trick annotations in a similar fashion, authortrees that track authors and groups of authors for different portions ofa document, and search results trees that track nested search results(i.e., with the root of the tree as an original search and child nodesbeing drill-down searches) are all examples of the different types ofobject trees possible using the multiple representation data modeldescribed herein.

III. Example Uses of the Multiple Representations

The above sections described examples of the multiple representationsfor a document used by some embodiments. As described in Sections I andII, some embodiments store (either as files or in-memoryrepresentations) a content stream and one or more object trees torepresent a document. These object trees may include a documentstructure tree, as described in Section I, and a style properties tree,as described in Section II, or other object trees. These differentrepresentations may be used in a complimentary fashion. The followingsub-sections will describe three different uses of the differentrepresentations: selecting a content object, navigating a document, andconverting the format of a document.

A. Selection

Selecting a content object (e.g., a word, text line, paragraph, etc.) isa process that can be performed very efficiently by using a combinationof the document structure tree and the content stream. Essentially, whena user of a document provides input to select a content object (e.g., aword, paragraph, etc.), the application displaying the document canassociate the user input with particular x-y coordinates of a particularpage, and then can identify the appropriate level node in the documentstructure tree that corresponds to the page and coordinates. This nodeindicates a range in the content stream at which the content is located.

FIG. 9 illustrates such a word selection process of some embodiments.This figure illustrates the page 200, and thus the document structuretree 400 along with content stream 300 generated for this document, asdescribed above in Section I. As shown at the left side of FIG. 9, acursor 905 is currently located over the word “pilowestan” in thedocument 200. In order to select this word, the user performs a wordselection operation (e.g., double-clicking a cursor controller, doubletapping a touchscreen over the display of the word, or a different wordselection input operation).

When the user enters this selection input, the application resolves theinput to a particular x-y coordinate and determines the type of contentstructure the user wishes to select. As mentioned, in the illustratedcase the user has provided input for selecting a word. In some cases, auser will select a text line or a paragraph (e.g., using a triple-clickor triple-tap, a common input for paragraph selection). The user mightalso hold down and drag the cursor (or finger along a touchscreen), inwhich case the start point of the selection will need to be resolved toa particular character.

As shown in the center of FIG. 9, this selection leads the applicationto navigate the object tree 400 to identify the word at the selectedcoordinate. The object tree is traversed from the page level to thecolumn level (each of which only include one node), then the paragraphnodes are searched in order until the paragraph whose bounds include theidentified coordinate is found. Within the identified paragraph node,the text line nodes are searched until a text line whose bounds includethe identified coordinate is found. This process is repeated at the wordlevel by searching all children of the identified text line (the wordsin the text line) to find the bounds of the word being selected.

As shown, in this case a word node 910 is identified as the selectednode. This word object stores an index range of 780-800, referencing thecontent stream 300. This is indicated in FIG. 9 by the arrow that pointsfrom the node 910 into content stream 300. The hex code characters atthese indices are then identified in the stream. Because the indices areknown, there is no need to read from the beginning of the stream, so thecharacters can be identified quickly.

The application displaying the document can then highlight thecharacters in the selected word in order to indicate the selection. FIG.10 illustrates the page 200 with the word 1005 highlighted. Someembodiments perform this highlighting by applying an image filter to thedisplay of the characters or by modifying the styling properties of thecharacters.

From this point, the user may extend this selection by dragging aninsertion marker (e.g., the cursor, a finger on a touchscreen, etc.) orusing keystrokes (e.g., shift+arrow). Because the stream representationis ordered, as the selection range includes additional characters, thestart index of the range can be decremented (or the end indexincremented) to include these additional characters and appropriatelymodify the display of the characters.

If the user performs additional operations on the selected text, theapplication may switch back to using one or both of the objectrepresentations. For example, if the user copies the selected text to avirtual clipboard, then the application may use the objectrepresentation to convert the format of the selected portion, asdescribed in detail below in subsection III.C. If the user applies astyle property to the text (e.g., underlining the text), then the styleproperty object tree will be modified.

Whichever tree (i.e., set of hierarchically-related objects) anapplication accesses, the application can find the necessary objectseasily because the index ranges of the objects (either content objectsor style objects) are already known, and the constraints on the objecttree ensure that adjacent characters occur in adjacent leaf nodes(adjacent in a depth-first ordering of the tree). For instance, if thestart index is decremented by two, then the selection will haveincreased to include a character in “tageneutian”, which is the lastdescendant of the prior sibling of the parent node of the “pilowestan”word node. That is, the application would determine that there is noprior sibling of the word node 910 (because it is the first word in thetext line), and thus would move up to the prior sibling of the text linenode 439. This is the text line node 438 (stream indices 679-778), andthe application would then find the last descendant of this node.

B. Navigation

In certain situations, when a user navigates a document, the combinationof the content stream representation and object tree representation fora document may be used for more efficient processing. The combination ofthe two representations proves especially useful when a user chooses tonavigate to a specific portion of a document and then move around toadjacent portions. These processes may be especially applicable on smallscreen devices (e.g., smart phones) in which a user is less likely towant to navigate by scrolling through a document, but are equallyapplicable to applications operating on any electronic device.

FIG. 11 conceptually illustrates a process 1100 of some embodiments foridentifying a desired object in a document structure tree thatrepresents a structure in the document to which a user wishes tonavigate. The process 1100 will be described in part by reference toFIG. 12, which illustrates a particular example of such a process. InFIG. 12, a user searches for a particular chapter title in an e-book1200, and the application uses the document structure tree 1205 for thee-book to identify an index range in the content stream 1210 for thee-book. In some embodiments, the process 1100 is performed by anapplication for displaying a document (e.g., the e-book) such as a PDFviewer, a word-processing application, an e-book reader, etc.

As shown, FIG. 12 illustrates that two representations are stored fore-book 1200. The first representation is a content stream representation1210 that includes representations (e.g., in hex code) for all of thecharacters in the book, including the chapter titles. The secondrepresentation is a document structure tree 1205 (i.e., a set ofhierarchically-arranged objects). The objects in the document structuretree divide the book into chapters, the first child of which is always atitle object. Following the title are the paragraphs of the particularchapter. In some embodiments, the title object is a specific type ofparagraph object (e.g., a paragraph object with a variable indicatingthat it is a title).

Many of these nodes would have child nodes (text lines, primitivecharacters, etc.). One of ordinary skill in the art will recognize thatother document structure models are possible for such a document. Forinstance, the child nodes of the chapter nodes could be page nodes, andthe titles might then be the first child node of the first page node ofeach chapter. However, as an e-book might be designed for display on anyof a variety of devices at different resolutions, pages might not be auseful construct for its representation. Instead, an application wouldidentify the content in a chapter (using the stream indices for thechapter and its child objects) and then display that content accordingto the constraints of the display.

In FIG. 11, as shown, the process 1100 begins by receiving (at 1105) aspecification of the level of a target node. In the case of FIG. 12,this target is the title node. The application might receive thespecification of the target when a user selects an option presented inthe application to search for a title (e.g., by selecting a title optionfor a search box). More generally, the level of the target may beidentified from a direct indication by a user or from contextual clues.As indicated by the example, the specification of the searched—forobject may include more than just the level of the object in thestructure hierarchy, but also its place in the ordering (e.g., in thiscase, the first sibling at its level). The process also receives (at1110) a description of the target node. In some embodiments, thisdescription may be the text the user wishes to find (e.g., “Chapter 2Title”). For instance, a user could input specific text into a searchbox and then press a “search” user interface item.

The process 1100 then retrieves (at 1115) an index range from a node ofthe object tree that meets the specifications. In some embodiments, theprocess traverses the tree in order to find the first such node. Asmentioned, these specifications might include a hierarchical level ofthe sought-after object as well as additional descriptors (e.g., thatthe object be a first sibling at the specified level). For instance, inthe case of FIG. 12, the first node retrieved would be node 1215, thenode for the first chapter title, as this is the first node at theparagraph level that is a first sibling. The process then uses (at 1120)the retrieved index range to retrieve content from the stream. That is,the process retrieves the content from the stream at the indicesspecified by the identified node. In this case, the retrieved content isthe title of chapter 1.

The process determines (at 1125) whether the content retrieved from thestream meets the target description that was received at 1110. Someembodiments compare the character values of the retrieved content tocharacter values of the desired content to determine whether the twomatch. In the example of FIG. 12, the first chapter title would notmatch up with the searched content (unless, of course, the first andsecond chapters have the same title).

When the two character values do not match, the process retrieves (at1130) another index range from another node of the object tree thatmeets the specifications, then returns to 1120. Thus, the process usesthe object tree to quickly search, in order, the content of only theobjects that match the specifications indicated by the user. In theexample situation, the process would move to the first child of thesecond chapter object (node 1220) and retrieve the index range for thesecond chapter title. In this case, the process would find a matchbetween the searched title terms and the chapter title for chapter 2.

When a match is found, the process reports (at 1135) the location of thetarget node. The application can then perform whatever action isrequired using the identified target object. In the example of searchingfor a chapter title, the application will generally want to do more thanjust display the title. The application may access the stream using theindex range of the entirety of chapter 2 and display as much of theretrieved content as will fit in the display area for the e-book, givena font size and style chosen by the user. Other embodiments (e.g., forother uses) will display content on either side of the identified targetnode (e.g., centering the content of the target node in the display),and possibly highlight the target itself in the display. In addition,some embodiments may perform non-display functions, such as copying thecontent of the target object to a virtual clipboard, storing a markerreferencing the target object, etc.

C. Format Conversion

When a document-processing application copies a selection to apasteboard (often called a clipboard) or otherwise passes on a portionof a document to another application, it is often necessary to convertthe document to a common format understood by both applications. Forexample, semantic reconstruction on a PDF document will produce anobject tree designed for the task (e.g., as described in US2010/0174985, incorporated by reference above), but when copying aselection to the pasteboard, the selection is converted to HTML, RTFand/or plain text to ensure various other applications can use thatinformation. Export of the document to a user-selected format is just aspecial case of this conversion procedure in which the selection toconvert is the entire document.

In some embodiments of the multiple representation data model, aselection is represented as a sequence of non-overlapping ranges ofindices into the content stream, ordered by start index. The selectionmight be the result of a text search or other query, or might resultfrom the user indicating regions on the screen (e.g., as described abovein Section I.A). To convert a selection to another format, it isnecessary to identify the nodes that participate in the selection,traverse them in an appropriate order, and append the appropriate outputfor each node. The conversion of a node is generally dependent on itstype and may include an initial output, followed by the conversion ofits children that are included in the selection, and then followed by afinal output.

FIG. 13 conceptually illustrates a process 1300 for using the dualrepresentation data model to convert a selection of a portion of thedocument into another format. FIG. 13 will be described by referencingexamples illustrated in FIGS. 14-15. FIG. 14 illustrates a selection inan example document page 1400. FIG. 15 illustrates an example objecttree 1510 for the document page 1400. The object tree 1510 includes pagelevel root node 1520, paragraph level child nodes 1530-1532 and wordlevel grandchild nodes 1541-1548. As mentioned, different embodimentsuse different levels of nodes in the document structure object tree(e.g., column nodes, text line nodes, etc.).

FIG. 15 also illustrates an example output stream 1550 in a new formatproduced by the conversion process using the object tree 1510. Theoutput stream 1550 includes sections 1551-1563 that correspond to nodesin the object tree 1510. The selection range 1405 covers indices 185 to468 in the content stream, which are reference by the highlighted nodesin the object tree 1510 (nodes 1520, 1530, 1532, 1534, and 1543-1547).Each of the nodes that overlap the selection range causes one or moresections of data to be appended to the output stream 1550 (e.g., node1532 causes sections 1555 and 1558 to be appended into the outputstream).

As shown, the process 1300 of FIG. 13 begins when it receives (at 1301)a selection of a portion of a document and an indication that theselection needs to be converted (e.g., a copy operation input by a user,instructions to export the document, etc.). As illustrated in theexample of FIG. 14, a user has selected a selection 1405 (highlightedportion) of the document page 1400 from the last word “comptory” of thefirst paragraph to the second word “strule” of the third paragraph, andcopied this selection (e.g., with “Ctrl-C” or “Cmd-C” on a keyboard, amenu selection, etc.). The selection could also be the entire documentif the user chose to export the document. In addition, various otheroperations could result in the conversion of a portion or entirety of adocument.

In some embodiments, such a selection is represented as a sequence ofnon-overlapping selection ranges R₁, R₂ . . . , R_(K), where the rangeof any R_(i) is before the range of R_(i+1) in the content stream. Aselection may include multiple non-overlapping selection ranges when aselection is disjoint (e.g., when the selection is a set of searchresults for paragraphs including a particular phrase). The selectionranges are used for comparison with ranges of indices stored in thenodes of the object tree for the document containing the selection. Eachcomparison is performed between a current selection range R_(i) and acurrent node N. Some embodiments keep and update a pointer to a currentselection range R_(i) and a pointer to a current node N in the objecttree throughout the operation of the process 1300.

The process 1300 next makes (at 1302) the root node of the object treethe current node N. In the example illustrated in FIG. 15, the node 1515in the object tree 1510 is the root node. This node holds the range ofindices that covers the entire document page 1400. The process 1300makes (at 1304) the first selection range R₁ in the sequence ofselection ranges the current selection range.

After making the root node the current node and the first selectionrange the current selection range, the process converts (at 1306) thesequence of selection ranges into a new format. In some embodiments,this operation recursively traverses nodes of an object tree of the datamodel, starting with the root node as the current node and the firstselection range as the current selection range. After the completion ofthe conversion, the process 1300 ends. FIGS. 16 a-16 b conceptuallyillustrates an example process 1600 of some embodiments that performssuch a recursive conversion operation. The process 1600 may be performedat operation 1306 of the process 1300 in some embodiments. FIGS. 16 a-16b will also be described by referencing the examples illustrated inFIGS. 14-15.

The process 1600 is a recursive process that recursively traverses nodesof an object tree in order to convert the sequence of selection ranges.The process 1600 is recursive because it includes an operation thatperforms the process 1600 itself. The process 1600 inherits a currentnode and current selection range that have been previously set (e.g., byan operation such as operation 1300 or by a previous instance of process1600 that recursively calls a current instance of process 1600. Theprocess 1600 also inherits values or pointers saved from a previousinstance of the process, such as a saved node or a flag for indicatingcompletion of the conversion process.

The process 1600 starts by comparing (at 1610) the current selectionrange with the range of the current node. If the range of indices in thecurrently selected node overlaps the current selection range (that is,the content referenced by the currently selected node is part of theselected content in the current selection range), then the process 1600proceeds to 1612 to append data to an output in a new format. If therange of indices in the current selection node N does not overlap thecurrent selection range the process 1600 proceeds to 1620.

At 1620, the process determines whether the current selection rangeR_(i) is before the range of indices stored in the current node N. Sincethe sequence of selection ranges R₁ . . . R_(N) is arranged in ascendingorder, the process will go to the next selection range R_(i+1) in thesequence to look for an overlap between the selection range and thecurrent node. However, if the current selection range is after the rangeof indices of the current node, then the process must wait for the nextnode in the object tree in order to look for an overlap with theselection range. If the current selection range R_(i) is before therange in the current node N, the process 1600 proceeds to 1630 in orderto move on to the next selection range R_(i+1). If the current selectionrange R_(i) is not before the range of the current node (i.e., thecurrent selection range is after the range of the current node), thenthe process returns to resume a previous level of the recursiveoperation.

At 1630, the process determines whether the current selection rangeR_(i) is the last selection range R_(K). Some embodiments make thisdetermination by examining a pointer to a current selection range anddetermining whether this points to the last range. When the currentselection range is the last selection range in the sequence, then theselection ranges have been completely converted, as the process hastraversed the object tree to the point that the current node referencescontent starting after all of the selection ranges. The certainty ofthis determination is made possible by the ordering and completenessconditions imposed on the object tree described in Section I above. Inthis case, the process 1600 proceeds to 1635 to report that theconversion of the selection ranges is done. On the other hand, if thecurrent selection range is not the last selection range, the processproceeds to 1640.

When the process has converted all of the selection ranges, the processreports (at 1635) that the conversion of the selection range is done. Insome embodiments, this is accomplished by setting a “done” flag. Oncethis flag is set, the process 1600 knows that the entire selection hasbeen converted, and that there is no need to further traverse any nodein the object tree. After reporting that the conversion of the selectionrange is done, the process returns to resume a previous level ofrecursive operation (i.e., the various instances of the process 1600 canbegin ending).

When more selection ranges remain to be converted, the process makes (at1640) the next selection range R_(i+1) the current selection range. Insome embodiments, this is accomplished by updating a pointer for thecurrent selection range to reference R_(i+1) rather than R_(i). Afterupdating the current selection range, the process proceeds to 1610 todetermine whether the new current selection range overlaps the range inthe current node.

When the process 1600 determines that the current selection range doesoverlap the range referenced by the current node, the process appends(at 1612) an initial output to the output stream according to the newformat (i.e., the format to which the selection is being converted).This initial output is data that the process appends into the outputstream before traversing down into the offspring nodes of the currentnode. In some embodiments, this initial output is determined by thedesired output format for the type of node. For example, if the newformat requires the start of every paragraph to include a tab character,then a paragraph level node that overlaps the selection range will haveto append a tab character into the output stream. Often, the initialoutput will include tags—for instance, if the output is HTML, then eachparagraph will start with the HTML tag for starting a paragraph (i.e.,“<p>” as well as any attributes of the paragraph that need to be set).In the example illustrated in FIG. 15, when the process 1600 initiallyencounters the paragraph level node 1532 for a format that requires aparagraph to start with a tab character, the process 1600 will append atab character to the output stream 1550 a part of the section 1555.

After appending the initial output, the process saves (at 1614) thecurrent node. As the process 1600 is a process that will recursivelytraverse through offspring nodes of the current node in the object tree,some embodiments save the current node (e.g., by “pushing” the pointerto the current node into a last-in-first-out stack data structure) sothe current node can be restored after the traversal of its offspringnodes is complete. After saving the current node, the process proceedsto 1650.

The process then determines (at 1650) whether the current node has anychild nodes. In some embodiments, a node without a child node is a leafnode in the tree structure. If the current node is a leaf node that doesnot have a child node, the process proceeds to 1655 to retrieve andoutput the content associated with the leaf node. On the other hand,when the current node has at least one child node, the process proceedsto 1660 to traverse each of the child nodes of the current node.

Next, the process retrieves (at 1657) content associated with thecurrent node from the ordered content stream for the document. In someembodiments, the process uses the indices stored in the current node forretrieving corresponding data from the stream representation. In theexample of FIG. 15, the process uses the index range 451-458 stored inthe word level leaf node 1545 to retrieve the word “twight” from astream representation of the document page 1400. After retrievingcontent from the stream representation, the process proceeds to 1657.

The process then appends (at 1657) the content retrieved from the streamrepresentation to the output stream. In the example of FIG. 15, theprocess appends the retrieved word “twight” to the output stream at1657. In some embodiments, the process 1600 appends this retrievedcontent to the output stream after the initial output for the node(appended at 1612) and before the final output for the node (appended at1680). Since only leaf nodes without child nodes perform operations 1650and 1657, the process 1600 retrieves content from the streamrepresentation and appends content to the output stream only for leafnodes. After appending contents to the output stream, the processproceeds to 1680.

In some embodiments, the process retrieves content from the streamrepresentation for both leaf nodes and non-leaf nodes. During the courseof generating the output for a non-leaf node, some embodiments determinewhether there is a gap between the ranges of indices of the child nodesof the non-leaf node. As discussed in Section I above, in someembodiments a strict rule is applied such that the range of indices of aparent node must equal the union of the ranges of its child nodes (i.e.,there are no gaps in the index ranges of the child nodes). However, inother embodiments, this condition is relaxed such that the index rangeof a parent node merely contains the union of the ranges of its childnodes such that there may be gaps in index ranges between child nodes.In some such embodiments, the process 1600 determines whether there is agap between the previous index range for which content is retrieved (ate.g., 1655 or 1657) and the next node, and that this gap is a gap withinthe parent node (as opposed to a gap in the content that should beconverted). If there is such a gap, then the process retrieves contentcorresponding to the index range of the gap from the content stream andappends the retrieved content into the output stream.

When appending content into the output stream, instead of appending thecontent retrieved from the stream representation directly to the outputstream, some embodiments add additional content or transform the contentaccording to the specified output format. For example, in someembodiments that include a style tree as discussed earlier in SectionII, the process transforms the content to be appended to the outputstream according to nodes of the style tree. In some of theseembodiments, the process finds nodes in the style tree that intersectthe range of the current node and applies the style to the content ofthe current node according to the range specified in these overlappingstyle tree nodes. In some embodiments in which the range of indices in aparent node is the union of the range of indices in the child nodes, theprocess examines all leaf nodes in the style tree for determining thestyle of the content to be appended. In some other embodiments in whichleaf nodes do not encompass the entire range of indices but insteadleaves “gaps” between indices ranges as discussed earlier in Section II,the process deduces the style for content that falls in the “gaps” byexamining the style of the parent node in the style tree.

When there is at least one child node of the current node, the processsets (at 1650) the first child node of the current node as the newcurrent node. The first child node is the child node that contains thelowest range of indices of its siblings. In some embodiments, childnodes are arranged in an array ordered by starting index, and the firstchild node can be identified as the first node in the array. In someembodiments, the process updates a current node pointer to point at thefirst child node.

Next, the process 1600 traverses (at 1670) a child node by recursivelyperforming the same selection conversion process for the child node(i.e., the new current node) by proceeding to 1610. This is a recursivecall that initiates a new level of recursive operation of process 1600(essentially restarting an instantiation of process 1600 that, whenfinished, will return to the current instantiation of the process (i.e.,1672). This new level of the recursive operation will return tooperation 1672 at the current level after the selection conversionprocess has been completed for the child node. One of ordinary skillwill recognize that the newly called process may recursively call yetanother level of the process if the child node being traversed has childnodes of its own, and that these child nodes may also call yet anotherlevel of the process, and so on, until the leaf nodes of the currentnode that are in the selection range have been traversed.

When a particular instantiation of process 1600 returns (from arecursive call to an instantiation that traverses a child node), theprocess determines (at 1672) whether the conversion of the selectionrange is done. As mentioned earlier, the process 1600 is complete whenthe selection has been completely converted. After the last selectionrange has been converted, the process 1600 in these instances stores anindication (such as by setting a “done” flag at 1635) to indicate thatthe selection conversion is done so the process no longer traversesadditional child nodes. If conversion of the selection range is done(i.e., a “done” flag is set), the process proceeds 1680. If not, theprocess proceeds to 1674.

When the selection range has not been completely converted, the processdetermines (at 1674) whether there are additional child nodes to betraversed under the current node. In some embodiments, the processdetermines that there are no more child nodes to be traversed when thecurrent node pointer currently references the last child node (e.g., thelast node in an array of child nodes. When all child nodes of thecurrent node have been traversed, the process proceeds to 1680.

On the other hand, when additional child nodes remain to be traversed,the process sets (at 1676) the next child node as the current node. Asstated, some embodiments arrange a set of sibling nodes (e.g., the childnodes of the current node) in an array so that the next child node canbe identified as the next node in the array. Having updated the currentnode to the next child node, the process proceeds to 1670 to traversethe next child node by recursively performing the same selectionconversion process.

Once the process 1600 has received indication that the conversion of theselection range is complete or that there are no more child nodes to betraversed, the process 1600 appends (at 1680) a final output into theoutput stream according to the desired new format. This final output isdata to be appended into the output stream for the current node afteroffspring nodes of the current node have all been traversed (or when theselection range has been completely converted). In some embodiments,this final output is determined by the desired output format for type ofnode. For example, if the new format requires the end of every paragraphto include carriage return, then a paragraph level node that overlapsthe selection range will have to append carriage return character intothe output stream. In the example illustrated in FIG. 15, when theprocess 1600 has finished traversing all offspring nodes of theparagraph level node 1532 for a format that requires a paragraph to endwith a carriage return character, the process 1600 will append acarriage return character to the output stream 1550 a part of thesection 1558.

Next, the process restores (at 1690) the most recently saved node as thecurrent node (e.g., by “popping” a saved pointer from thelast-in-first-out stack data structure). After restoring the currentnode to the previously saved node, the process returns to resume aprevious level of recursive operation. This previous level of recursiveoperation will be the instance in which the new current node (the nodejust popped from the stack) was the current node, from which the currentlevel of recursive operation was called.

In several places, the process 1600 is described as returning to resumea previous level of recursive operation. In some embodiments, theprocess 1600 resumes a previous level of recursive operation byproceeding to the operation performed right after the recursiveoperation. For example, the operation 1670 is an operation that performsthe process 1600 recursively. Upon completion of the operation 1670, theprocess 1600 then resumes at the next operation 1672. If the operation1600 is entered into from 1306 of FIG. 13, then the previous level ofrecursive operation is process 1300, which ends after 1306.

The description for processes 1300 and 1600 describes the conversion ofa selection of data from an object model on a stream into a new stream(e.g., an HTML file). Some embodiments use similar processes to performconversions between one object model on the character stream and adifferent object model on the same stream. In some embodiments, such aconversion process uses the initial and final outputs to define thenodes of the tree (e.g., in XML). Rather than append content from thecharacter stream in between the initial and final outputs, ranges(either the same set of ranges as the first object model or differentsets of ranges) may be used as information stored in the nodes of thenew object model.

Some embodiments use the XSLT programming language to perform suchtransformations, as this language allows for transforming between almostany pairs of object models. For instance, variations on the describedconversion process could be used to perform a query on a table (e.g.,presenting the table with the rows reordered according to a givencriteria). A second example would be to take an object tree and convertthe tree into graphics (e.g., an SVG object tree).

IV. Electronic System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or morecomputational or processing unit(s) (e.g., one or more processors, coresof processors, or other processing units), they cause the processingunit(s) to perform the actions indicated in the instructions. Examplesof computer readable media include, but are not limited to, CD-ROMs,flash drives, random access memory (RAM) chips, hard drives, erasableprogrammable read only memories (EPROMs), electrically erasableprogrammable read-only memories (EEPROMs), etc. The computer readablemedia does not include carrier waves and electronic signals passingwirelessly or over wired connections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storagewhich can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 17 conceptually illustrates an electronic system 1700 with whichsome embodiments of the invention are implemented. The electronic system1700 may be a computer (e.g., a desktop computer, personal computer,tablet computer, etc.), phone, PDA, or any other sort of electronicdevice. Such an electronic system includes various types of computerreadable media and interfaces for various other types of computerreadable media. Electronic system 1700 includes a bus 1705, processingunit(s) 1710, a graphics processing unit (GPU) 1715, a system memory1720, a network 1725, a read-only memory 1730, a permanent storagedevice 1735, input devices 1740, and output devices 1745.

The bus 1705 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 1700. For instance, the bus 1705 communicativelyconnects the processing unit(s) 1710 with the read-only memory 1730, theGPU 1715, the system memory 1720, and the permanent storage device 1735.

From these various memory units, the processing unit(s) 1710 retrievesinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments. Someinstructions are passed to and executed by the GPU 1715. The GPU 1715can offload various computations or complement the image processingprovided by the processing unit(s) 1710. In some embodiments, suchfunctionality can be provided using CoreImage's kernel shading language.

The read-only-memory (ROM) 1730 stores static data and instructions thatare needed by the processing unit(s) 1710 and other modules of theelectronic system. The permanent storage device 1735, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system1700 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 1735.

Other embodiments use a removable storage device (such as a floppy disk,flash memory device, etc., and its corresponding drive) as the permanentstorage device. Like the permanent storage device 1735, the systemmemory 1720 is a read-and-write memory device. However, unlike storagedevice 1735, the system memory 1720 is a volatile read-and-write memory,such a random access memory. The system memory 1720 stores some of theinstructions and data that the processor needs at runtime. In someembodiments, the invention's processes are stored in the system memory1720, the permanent storage device 1735, and/or the read-only memory1730. For example, the various memory units include instructions forprocessing multimedia clips in accordance with some embodiments. Fromthese various memory units, the processing unit(s) 1710 retrievesinstructions to execute and data to process in order to execute theprocesses of some embodiments.

The bus 1705 also connects to the input and output devices 1740 and1745. The input devices 1740 enable the user to communicate informationand select commands to the electronic system. The input devices 1740include alphanumeric keyboards and pointing devices (also called “cursorcontrol devices”), cameras (e.g., webcams), microphones or similardevices for receiving voice commands, etc. The output devices 1745display images generated by the electronic system or otherwise outputdata. The output devices 1745 include printers and display devices, suchas cathode ray tubes (CRT) or liquid crystal displays (LCD), as well asspeakers or similar audio output devices. Some embodiments includedevices such as a touchscreen that function as both input and outputdevices.

Finally, as shown in FIG. 17, bus 1705 also couples electronic system1700 to a network 1725 through a network adapter (not shown). In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 1700 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself. In addition, someembodiments execute software stored in programmable logic devices(PLDs), ROM, or RAM devices.

As used in this specification and any claims of this application, theterms “computer”, “server”, “processor”, and “memory” all refer toelectronic or other technological devices. These terms exclude people orgroups of people. For the purposes of the specification, the termsdisplay or displaying means displaying on an electronic device. As usedin this specification and any claims of this application, the terms“computer readable medium,” “computer readable media,” and “machinereadable medium” are entirely restricted to tangible, physical objectsthat store information in a form that is readable by a computer. Theseterms exclude any wireless signals, wired download signals, and anyother ephemeral signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. In addition, a number of the figures(including FIGS. 11, 13, and 16) conceptually illustrate processes. Thespecific operations of these processes may not be performed in the exactorder shown and described. The specific operations may not be performedin one continuous series of operations, and different specificoperations may be performed in different embodiments. Furthermore, theprocess could be implemented using several sub-processes, or as part ofa larger macro process. Thus, one of ordinary skill in the art wouldunderstand that the invention is not to be limited by the foregoingillustrative details, but rather is to be defined by the appendedclaims.

1-18. (canceled)
 19. A non-transitory machine readable medium storing aprogram for execution by at least one processing unit, the programcomprising sets of instructions for: receiving a document having a firstformat, the document stored as a stream representation comprising anordered content stream and an object representation comprising a set ofobject nodes arranged in a tree structure, the object nodes referencingranges of indices in the content stream; and converting an identifiedrange of the document into a new format by recursively traversing thetree structure to identify nodes referencing indexes within theidentified range and converting the nodes to the new format.
 20. Thenon-transitory machine readable medium of claim 19, wherein the objectnodes represent document structures.
 21. The non-transitory machinereadable medium of claim 19, wherein the set of instructions forconverting comprises a set of instructions for: receiving an object nodethat references a range within the identified range: outputting aninitial output associated with the object node; when the object node isa parent of at least one object node, recursively traversing any childobject nodes of the received object node; when the object node is not aparent of any object nodes, outputting content from the content streamat the range referenced by the object node; and upon returning from anyrecursive traversal of child nodes, outputting a final output associatedwith the object node.
 22. The non-transitory machine readable medium ofclaim 21, wherein the new format is an HyperText Markup Language (HTML)format, and the initial and final outputs comprise HTML tags.
 23. Thenon-transitory machine readable medium of claim 21, wherein the objectrepresentation is a first object representation, wherein the document isfurther stored as a second, different object representation comprising asecond set of object nodes arranged in a second tree structure, theobject nodes in the second object representation also referencing rangesof indices in the content stream.
 24. The non-transitory machinereadable medium of claim 23, wherein the second tree structure is astyle tree structure, wherein the set of instructions for convertingfurther comprises a set of instructions for transforming the outputtedcontent according to nodes in the style tree.
 25. The non-transitorymachine readable medium of claim 19, wherein the program furthercomprises a set of instructions for receiving a selection of a structurein the document to be converted into the new format, the selectioncomprising a plurality of non-overlapping ranges of indices in thecontent stream.
 26. The non-transitory machine readable medium of claim25, wherein the plurality of non-overlapping ranges is the identifiedrange for conversion.
 27. A method for converting a document into adifferent format, the method comprising: receiving a document having afirst format, the document stored as a stream representation comprisingan ordered content stream and an object representation comprising a setof object nodes arranged in a tree structure, the object nodesreferencing ranges of indices in the content stream; and converting anidentified range of the document into a new format by recursivelytraversing the tree structure to identify nodes referencing indexeswithin the identified range and converting the nodes to the new format.28. The method of claim 27, wherein the object nodes represent documentstructures.
 29. The method of claim 27, wherein converting theidentified range of the document into the new format further comprises:receiving an object node that references a range within the identifiedrange: outputting an initial output associated with the object node;when the object node is a parent of at least one object node,recursively traversing any child object nodes of the received objectnode; when the object node is not a parent of any object nodes,outputting content from the content stream at the range referenced bythe object node; and upon returning from any recursive traversal ofchild nodes, outputting a final output associated with the object node.30. The method of claim 29, wherein the new format is an HyperTextMarkup Language (HTML) format, and the initial and final outputscomprise HTML tags.
 31. The method of claim 29, wherein the range ofindices of any parent object node is the union of the ranges of indicesof its child object nodes.
 32. The method of claim 29, wherein the rangeof indices of any parent object node contains the union of the ranges ofindices of its child object nodes.
 33. The method of claim 29, whereinthe range of indices for any two sibling object nodes are disjoint. 34.The method of claim 29, wherein the new format is designated based on auser selection.
 35. The method of claim 29, wherein outputting thecontent from the content stream comprises transforming the content basedon styling requirements defined by a style tree of the document.
 36. Anon-transitory machine readable medium storing a program for executionby at least one processing unit, the program comprising sets ofinstructions for: receiving a document comprising a plurality ofprimitive elements; identifying an order for the primitive elements inthe document; storing a stream representation of the document, thestream representation comprising indexed representations of theprimitive elements in the identified order; and storing an objectrepresentation of the document, the object representation comprising ahierarchically-arranged set of object nodes that reference ranges ofindices in the stream representation.
 37. The non-transitory machinereadable medium of claim 36, wherein each object node represents astructure in the document, wherein the program further comprises sets ofinstructions for: receiving a selection of a structure in the document;traversing through the hierarchically arranged set of object nodes toidentify the object node for the selected structure; and identifying, inthe stream representation, the primitive elements represented at therange referenced by the identified object node.
 38. The non-transitorymachine readable medium of claim 37, wherein the program furthercomprises a set of instructions for displaying the document with theselected word highlighted.
 39. The non-transitory machine readablemedium of claim 37, wherein the selected structure is a word.
 40. Thenon-transitory machine readable medium of claim 36, wherein each objectnode represents a structure in the document, wherein the program furthercomprises sets of instructions for: receiving a search request for aparticular type of structure that includes a particular set of primitiveelements; traversing the hierarchically arranged set of object nodes toidentify object nodes representing the particular type of structure;comparing, for each identified object node, the primitive elementsrepresented in the stream at the range reference by the object node withthe particular set of primitive elements, until a matching object nodeis found.
 41. The non-transitory machine readable medium of claim 40,wherein the program further comprises a set of instructions fordisplaying a portion of the document associated with the structurerepresented by the matching object node.